Just as the ancients constructed mosaics from squares of colored glass or stone, a pixel art graphics artist selects the color of each pixel in a pixel art image from a color palette. The video game and computer hardware of the 1980's and 1990's imposed limitations that channeled graphics designers toward pixel art. Some of the best loved video game characters and graphics of all time were created using pixel art. While modern graphics artists generally have more flexibility, some have chosen to use pixel art for aesthetic or other reasons.
Display resolution has increased dramatically over the last twenty years. Early home video game systems had display resolutions on the order of about 250×250 pixels or less. Currently, high definition televisions have display resolutions of 1920×1080 pixels, and some modern computer displays have even higher display resolution. When one attempts to use higher resolution displays to display lower resolution pixel art, the result can be unsatisfactory. Whereas early lower resolution displays might have tended to blur pixels together to create an aesthetically pleasing effect, higher resolution displays tend to bring out the blocky (pixelated) constitution of pixel art. The resulting images are not aesthetically pleasing.
A common solution is to “upscale” the pixel art images for rendering on modern displays. This often means replacing each pixel in the original image with a block of higher resolution pixels, and performing some type of filtering to smooth the resulting image. Various techniques are known for performing such upscaling. See for example Kopf et al, “Depixelizing Pixel Art” (Microsoft Research 2011).
One example non-limiting Pixel Art Upscaler (“PAU”) is a process that creates, from an input picture, a picture with a 2×2 resolution increase. This algorithm works well on images that contain pixel art, namely hand-made low resolution images using palletized colors, that are characterized by flat colors and outlines. The purpose of such a PAU algorithm is to upscale the original picture, while trying to keep the flat colors and the intended hard edges and reducing or removing the staircase effect of the original picture. In particular, the goal of the PAU is to modify high frequencies in the image to make edges sharper.
This process is successful but has been identified to have a side effect of potentially modifying the brightness of certain areas compared to the original image—for example, to modify low frequency spatial components in a way that causes the upscaled image to exhibit local brightness changes. Such modified brightness can cause concerns regarding photosensitivity. Work has been done in the past to analyze video for photosensitivity (see for example HardingFPA software available from Cambridge Research Systems, Ltd.) but further improvements are possible and desirable.
The example non-limiting technology herein provides a filtering algorithm that compensates for brightness modification so that moving images including but not limited to video games that are already compliant with photosensitivity guidelines can benefit from PAU without losing their compliance. One advantage provided by the example non-limiting technology herein is that in the context of a brightness-compensated real time pixel upscaler, image streams tested for photosensitivity before upscaling do not need to be retested after upscaling. Meanwhile, the example non-limiting technology herein can be used in any context where real time or other flash or brightness compensation is desired irrespective of previous testing and/or image source.
In one example non-limiting implementation, the system analyzes the low spatial frequencies of the original image and of the upscaled filtered image. Differences will be observed in the low frequency components of the two images in the general case since the PAU filter as a side effect introduces low frequency changes. The example non-limiting implementation applies a modification to images produced by the PAU to attempt to match the brightness of the original images in the low frequency spectrum. Such matching can be accomplished for example by simple brightness subtraction on a pixel by pixel or other localized basis. From a viewer perspective (e.g., based on typical blurring visual effects), the original image and the modified filtered image will look the same—demonstrating that there is no low frequency brightness creep.
The example non-limiting technology herein uses a concept that could be called filter band limiting. It modifies any filter into a filter that will correctly preserve low spatial frequencies (i.e., the source and destination images will appear identical or nearly identical when blurred). This process can effectively be used to improve, or make photosensitive-safe, any image filter such as denoising filters, deblocking filters, deblurring filters or other filters.
The example non-limiting technology herein thus provides systems and methods that guarantee that, whatever artifacts are introduced by pixel art or other upscaling, they do not incur any global brightness modifications. The resulting images are very close to being as safe as the original images in terms of photosensitivity.
Furthermore, the example non-limiting safe filter can be implemented on a central processing unit, a graphics processing unit, using embedded logic, or using any other specific, specialized or general purpose hardware. Additionally, television sets or any other display devices could benefit from the example non-limiting safe filter after applying their own filters, such as for example upscaling, deinterlacing, and/or noise reduction. Such a filter can also be inserted in broadcast TV company production systems, or movie post production systems. Many other applications are possible.